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A GRAPHICAL USER INTERFACE USING DETABL-IN-CONTEXT FOLDING 

The invention relates to the field of computer graphics processing, and more specifically to a 
graphical user interface using detail-in-context folding. 

BACKGROUND OF THE INVENTION 

5 Display screens are the primary visual display interface to a computer. One problem with these 
visual display screens is that they are limited in size, thus presenting a challenge to user interface 
design, particularly when larger amounts of information is to be displayed. This problem is 
normally referred to as the "screen real estate problem". 

Well-known solutions to this problem include panning, zooming, scrolling or combinations 
10 thereof. While these solutions are suitable for a large number of visual display applications, 
these solutions become less effective where the visual information is spatially related, such as 
maps, newspapers and such like. In this type of information display, panning, zooming and/or 
scrolling is not as effective as much of the context of the panned, zoomed or scrolled display is 
hidden. 

'T- 
IS A recent solution to this problem is the application of "detail-in-context" presentation 
techniques. Detail-in-context is the magnification of a particular region of interest (the 'focal 
region" or "detail") in a data presentation while preserving visibility of the surrounding 
information (the "context"). This technique has applicability to the display of large surface area 
media, such as maps, on limited size computer screens including laptop computers, personal 
20 digital assistants ("PDAs"), and cell phones. 

in the detail-in-context discourse, differentiation is often made between the terms 
•'representation" and presentation". A representation is a formal system, or mapping, for 
specifying raw information or data that is stored in a computer or data processing system. For 
example, a digital map of a city is a representation of raw data including street names and the 
25 relative geographic location of streets and utilities. Such a representation may be displayed 
visually on a computer screen or printed on paper. On the other hand, a presentation is a spatial 
organization of a given representation that is appropriate for the task at hand. Thus, a 

presentation of a representation organizes such things as the point of view and the relative 

i. 
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emphasis of different parts or regions of the representation. For example, a digital map of a city 
may be presented with a region magnified to reveal street names. 

In general, a detail-in-context presentation may be considered as a distorted view (or distortion) 
of a portion of the original representation where the distortion is the result of the application of a 
5 "lens" like distortion function to the original representation. A detailed review of various detail- 
in-context presentation techniques such as Elastic Presentation Space may be found in a 
publication by Marianne S. T. Carpendale, entitled "A Framework for Elastic Presentation 
Space' 9 (Carpendale, Marianne S. T., A Framework for Elastic Presentation Space (Burnaby, 
British Columbia: Simon Fraser University, 1 999)), and incorporated herein by reference. 

10 In general, detail-in-context data presentations are characterized by magnification of areas of an 
image where detail is desired, in combination with compression of a restricted range of areas of 
the remaining information (i.e. the context), the result typically giving the appearance of a lens 
having been applied to the display surface. Using die techniques described by Carpendale, points 
in a representation are displaced in three dimensions and a perspective projection is used to 

IS display the points on a two-dimensional presentation display. In detail-in-context presentation 
systems, when a lens is applied to a two-dimensional continuous surface representation, for 
example, the resulting presentation appears to be three-dimensional. In other words, the lens 
transformation appears to have stretched the continuous surface in a third dimension. 

One shortcoming of present graphical user interfaces for operating systems and software 
20 applications is that user interface elements such as pop-up and drop-down menus comprised of 
lists of programs (i.e. in the case of operating systems) and function selections (i.e. in the case of 
software applications) often obscure or occlude other important information on the user's display 
screen including data, desktop icons, and other graphical elements. This is an example of the 
screen real estate problem referred to above. 

25 A need therefore exists for a graphical user interface for operating system and software 
application menus that avoids occlusion of displayed information. Consequently, it is an object 
of the present invention to obviate or mitigate at least some of the above mentioned 
disadvantages. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention may best be understood by referring to the following description 
and accompanying drawings. In the description and drawings, line numerals refer to like 
5 structures or processes. In the drawings: 

FIG. 1 is a graphical representation of the geometry for constructing a three-dimensional (3D) 
perspective viewing frustum, relative to an x, y, z coordinate system, in accordance with known 
elastic presentation space graphics technology; 

FIG. 2 is a graphical representation of the geometry of a presentation in accordance with known 
1 0 elastic presentation space graphics technology; 

FIG. 3 is a block diagram illustrating an exemplary data processing system for implementing an 
embodiment of the invention; 

FIG . 4 is a screen capture illustrating an operating system GUI in accordance with the prior art; 

FIG. 5 is a screen capture illustrating the operating system GUI of FIG. 4 with a selected pop-up 
IS menu in accordance with the prior art; and, 

FIG. 6 is a screen capture illustrating an operating system GUI using detail-in-context folding in 
accordance with an embodiment of the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description, numerous specific details are set forth to provide a thorough 
20 understanding of die invention. However, it is understood that the invention may be practiced 
without these specific details. In other instances, well-known software, circuits, structures and 
techniques have not been described or shown in detail in order not to obscure the invention. In 
the drawings, like numerals refer to like structures or processes. 

The term "data processing system" is used herein to refer to any machine for processing data, 
25 including the computer systems and network arrangements described herein. The term "Elastic 
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Presentation Space" ("EPS") (or "Pliable Display Technology" ("POT")) is used herein to refer 
to techniques that all w for the adjustment of a visual presentation without interfering with the 
information content of the representation. The adjective "elastic" is included in the term as it 
implies the capability of stretching and deformation and subsequent return to an original shape. 
5 EPS graphics technology is described by Carpendale in "A Framework for Elastic Presentation 
Space" (Carpendale, Marianne S. T., A Framework for Elastic Presentation Space (Bumaby, 
British Columbia: Simon Fraser University, 1999)), which is incorporated herein by reference. In 
EPS graphics technology, a two-dimensional visual representation is placed onto a surface; this 
surface is placed in three-dimensional space; the surface, containing the representation, is viewed 
10 through perspective projection; and the surface is manipulated to effect the reorganization of 
image details. Hie presentation transformation is separated into two steps: surface manipulation 
or distortion and perspective projection. 

FIG. 1 is a graphical representation 100 of the geometry for constructing a three-dimensional 
("3D") perspective viewing frustum 220, relative to an x, y, z coordinate system, in accordance 
with known elastic presentation space (EPS) graphics technology. In EPS technology, detail-in- 
context views of two-dimensional ("2D") visual representations are created with sight-line 
aligned distortions of a 20 information presentation surface within a 3D perspective viewing 
frustum 220. In EPS, magnification of regions of interest and the accompanying compression of 
the contextual region to accommodate this change in scale are produced by the movement of 
regions of the surface towards the viewpoint ("VP") 240 located at the apex of the pyramidal 
shape 220 containing the frustum. The process of projecting these transformed layouts via a 
perspective projection results in a new 2D layout which includes the zoomed and compressed 
regions. The use of the third dimension and perspective distortion to provide magnification in 
EPS provides a meaningful metaphor for the process of distorting the information presentation 
surface. The 3D manipulation of the information presentation surface in such a system is an 
intermediate step in the process of creating a new 2D layout of the information. 

FIG. 2 is a graphical representation 200 of die geometry of a presentation in accordance with 
known EPS graphics technology. EPS graphics technology employs viewer-aligned perspective 
projections to produce detail-in-context presentations in a r ef erence view plane 201 which may 
30 be viewed on a display. Undistorted 2D data points are located in a basal plane 210 of a 3D 

4 
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perspective viewing volume or frustum 220 which is defined by extreme rays 221 and 222 and 
the basal plane 210. The VP 240 is generally located above the centre point of the basal plane 
210 and reference view plane ("RVP") 201. Points in the basal plane 210 axe displaced upward 
onto a distorted surface 230 which is defined by a general 3D distortion function (i.e. a detail-in- 
S context distortion basis function). The direction of the viewer-aligned perspective projection 
corresponding to the distorted surface 230 is indicated by the line FPo - FP 231 drawn from a 
point FPo 232 in the basal plane 210 through the point FP 233 which corresponds to the focus or 
focal region or focal point of the distorted surface 230. 

EPS is applicable to multidimensional data and is well suited to implementation on a computer 

10 for dynamic detail-in-context display on an electronic display surface such as a monitor. In the 
case of two dimensional data, EPS is typically characterized by magnification of areas of an 
image where detail is desired 233, in combination with compression of a restricted range of areas 
of the remaining information (i.e. the context) 234, the end result typically giving the appearance 
of a lens 230 having been applied to the display surface. The areas of the lens 230 where 

IS compression occurs may be referred to as the "shoulder" 234 of the lens 230. The area of the 
representation transformed by the lens may be referred to as the "lensed area". Hie leased area 
thus includes the focal region and the shoulder. To reiterate, the source image or representation 
to be viewed is located in the basal plane 210. Magnification 233 and compression 234 are 
achieved through elevating elements of the source image relative to the basal plane 210, and then 

20 projecting the resultant distorted surface onto the reference view plane 201. EPS performs detail- 
in-context presentation of n-dimensional data through the use of a procedure wherein die data is 
mapped into a region in an (n+1) dimensional space, manipulated through perspective 
projections in the (n+1) dimensional space, and then finally transformed bade into n-dimensional 
space for presentation. EPS has numerous advantages over conventional zoom, pan, and scroll 

25 technologies, including the capability of preserving the visibility of information outside 234 the 
local region of interest 233. 

For example, and referring to FIGS. 1 and 2, in two dimensions, EPS can be implemented 
through the projection of an image onto a reference plane 201 in the following manner. The 
source image or representation is located on a basal plane 210, and those regions of interest 233 
30 of the image for which magnification is desired are elevated so as to move them closer to a 
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reference plane situated between the reference viewpoint 240 and the reference view plane 201. 
Magnification of the focal region 233 closest to the RVP 201 varies inversely with distance from 
die RVP 201. As shown in FIGS. 1 and 2, compression of regions 234 outside die focal region 
233 is a function of both distance from the RVP 201, and the gradient of the function describing 
5 the vertical distance from the RVP 201 with respect to horizontal distance from the focal region 
233. The resultant combination of magnification 233 and compression 234 of the image as seen 
from the reference viewpoint 240 results in a lens-like effect similar to that of a magnifying glass 
applied to the image. Hence, the various functions used to vary the magnification and 
compression of the source image via vertical displacement from die basal plane 210 are 
10 described as lenses, lens types, or lens functions. Lens functions that describe basic lens types 
with point and circular focal regions, as well as certain more complex lenses and advanced 
capabilities such as folding, have previously been described by Carpendale. 

System. FIG. 3 is a block diagram of an exemplary data processing system 300 for implementing 
an embodiment of the invention. The data processing system is suitable for implementing EPS 

15 technology and for viewing detail-in-context presentations in conjunction with a graphical user 
interface ("GUI"). The data processing system 300 includes an input device 310, a central 
processing unit or CPU 320, memory 330, and a display 340. The input device 310 may include 
a keyboard, mouse, trackball, or similar device. The CPU 320 may include dedicated 
coprocessors and memory devices. The memory 330 may include RAM, ROM, databases, or 

20 disk devices. And, the display 340 may include a computer screen or terminal device. The data 
processing system 300 has stored therein data representing sequences of instructions which when 
executed cause the method described herein to be performed Of course, the data processing 
system 300 may contain additional software and hardware a description of which is not 
necessary for understanding the invention. 

25 GUI Using Detail-In-Context Folding. FIG. 4 is a screen capture illustrating an operating system 
GUI in accordance with the prior art FIG. 5 is a screen capture illustrating the operating system 
GUI of FIG. 4 with a selected pop-up menu in accordance with the prior art As mentioned, one 
shortcoming of present graphical user interfaces for operating systems and software applications 
is that user interface dements such as pop-up and drop-down menus comprised of lists of 

30 programs (i.e. in the case of operating systems) and function selections (i.e. in the case of 
i 
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software applications) often obscure or occlude other important information on the user's display 
screen including data, desktop icons, and other graphical dements. For example, the appearance 
of the appearance of the pop-up "Start Menu" in FIG. 5 has occluded the visibility of desktop 
i ons from view. 

5 PDT and EPS technology includes a concept, which may be referred to as "folding!", in which an 
in-context detail view of information can be displaced so as to move the region of interest within 
the plane of the display, while retaining a continuous connection with some of the contextual 
information. According to one embodiment of the present invention, folding is applied to the 
GUI of an operating system or software application to displace the occluding user interface 
10 elements and expose the underlying information or graphical elements. 

FIG. 6 is a screen capture illustrating an operating system GUI using detail-in-context folding in 
accordance with an embodiment of the invention. In FIG. 6, the operating system is Microsoft 
XP. A copy of the unoccluded desktop from FIG. 4, prior to invoking the Start Menu illustrated 
in FIG. 5, has been retained to bring back the original icons for display. 

IS In accordance with another embodiment of die invention, folding can be applied to a drop-down 
(or pull-down) menu in the user interface for an application to prevent or remove occlusion of 
data in the program. 

Data Carrier Product. The sequences of instructions which when executed cause the method 
described herein to be performed by the exemplary data processing system of FIG. 3 can be 
20 contained in a data carrier product according to one embodiment of die invention. This data 
carrier product can be loaded into and run by the exemplary data processing system of FIG. 3. 

Computer Software Product. The sequences of instructions which when executed cause the 
method described herein to be performed by die exemplary data processing system of FIG. 3 can 
be contained in a computer software product according to one embodiment of the invention. This 
25 computer software product can be loaded into and run by the exemplary data processing system 
of FIG. 3. 

Integrated Circuit Product. The sequences of instructions which when executed cause the 

r 

method described herein to be performed by the exemplary data processing system f FIG. 3 can 
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be contained in an integrated circuit product including a coprocessor or memory according to one 
embodiment of the invention. This integrated circuit product can be installed in the exemplary 
data processing system of FIG. 3. 
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